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ELECTRONIC MAIL METHOD AND SYSTEM USING 
ASSOCIATED AUDIO AND VISUAL TECHNIQUES 

BACKGROUND OF THE INVENTION 

Field of the Invention 

The invention relates to electronic mail methods and systems and more particularly to 
electronic mail methods and systems using associated audio and visual techniques. 

Background Information 

Over the past decade, the use of electronic mail (email) has become a ubiquitous form of 
communication. Email has become among the primary methods of communications used in both 
business and social settings. Some of the advantages of communication by email are the ability 
to transmit messages instantly, the ability to transmit graphics, the ability to easily send messages 
to a large number of recipients, etc. Because of these (and other) advantages, transmitting 
messages by email is more efficient than other types of communication modes, such as 
telephone, facsimile, and the like. 

Nevertheless, transmitting messages by email limits a sender's ability to convey emotions 
that may be associated with a message. For example, when sending a message, senders may 
wish to convey that they are angry, sad, happy, etc. Because the content transmitted by email is 
usually in a text format, it is difficult for a sender to convey emotions along with a message. Of 
course, certain punctuation (such as an exclamation point) or an emoticon may aid in conveying 
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a sender's emotion, but the value of using these symbols are limited. For example, it is difficult 
to use symbols to convey humor, impatience, urgency, empathy, etc. 

U.S. Patent No. 6,064,383 to Skelley entitled "Method And System For Selecting An 
Emotional Prosody For A Graphical Character" is directed to a user interface element which 
5 enables a user to select an appearance for a character that corresponds with an emotion and an 
emotional intensity. Nevertheless, Skelley and the other existing prior art do not permit a full 
range of capability of expressing senses or emotions associated with a message transmitted over 
a network. For example, users are typically limited to a fixed number of characters, and are 
unable to easily display a likeness that resembles the user in a lifelike manner (with respect to 

l(t\ audio and visual aspects) when sending such messages. Moreover, users cannot easily enter and 

pi 

Jg choose among message content formats, be it text-based, audio, visual, etc., particularly with 
Lfl respect to the user's own likeness. Further, because transmitting messages by email has become 
* so ubiquitous, it is desirable to have the ability to prepare such messages using a variety of 
f7 devices and input techniques. Therefore, there is the need to devise a method and system in 
\%l which a user can easily compose an email message with the ability to associate one or more 

multimedia components related to the message content. Additionally, there is the need to devise 
such a method and system wherein the user is not restricted to specific character choices, 
emotional displays, message formats or limited inputting techniques. 

20 SUMMARY OF THE INVENTION 

Accordingly, it is an object of the invention to provide a multi-mail system and 
method in which a sender may convey and a recipient can realize emotional aspects associated 
with substantive content of an email message by receiving a message that is more than textual in 
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nature. Voice recognition technology and programmatic relation of sound and graphics may be 
used to produce a talking image. In one embodiment, the image may include the user's own 
visual and/or audio likeness. In an alternate embodiment, the image may comprise any available 
visual and/or audio display selected by the user. 

The multi-mail message may be inputted by a user in a text format and transposed 
into a format including the selected image and/or voice. In an alternate embodiment, a spoken 
message may be converted into a format including the selected image and/or voice. The 
formatted messages are then stored and/or transmitted via an email system or some other 
electronic network. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing features and other aspects of the invention are explained in the following 
description taken in conjunction with the accompanying figures wherein: 

FIG. 1 A illustrates a network for supporting transmission of email messages according to 
one embodiment of the present invention; 

FIG. IB illustrates one embodiment of the central controller used in the system shown in 
Fig. 1A; 

FIG. 2 illustrates the login process according to one embodiment of the invention; 

FIG. 3 illustrates the login process according to a second embodiment of the invention; 

FIG. 4 illustrates the graphics and audio management process according to one 
embodiment of the invention; 

FIG. 5 illustrates the process of uploading an image or audio file according to one 
embodiment of the invention; 
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FIG. 6 illustrates the process of deleting an image or audio file according to one 
embodiment of the invention; 

FIG. 7 illustrates the process of manipulating an image on a website according to one 
embodiment of the invention; 

FIG. 8 illustrates the graphic add-on process according to one embodiment of the 
invention; 

FIG. 9 illustrates the sound effect process according to one embodiment of the invention; 

FIG. 10 illustrates the process of setting up or changing voice sample preferences 
according to one embodiment of the invention; 

FIG. 1 1 illustrates the process of creating a new message according to one embodiment 
of the invention; 

FIG. 12 illustrates the process of creating an audio portion of a message according to one 
embodiment of the invention; 

FIG, 13 illustrates the image processing process according to one embodiment of the 
invention; 

FIG, 14 illustrates the email creation process according to one embodiment of the 
invention; 

FIG. 15 illustrates the text to audio synthesis process according to one embodiment of the 
invention; 

FIG, 16 illustrates the process of receiving a spoken message by telephone according to 
one embodiment of the invention; and 

FIG, 17 illustrates the sound quality processing process according to one embodiment of 
the invention. 



574235_4 



4 



4056-4000 



It is to be understood that the drawings are for illustration only and are not limiting, 

DETAILED DESCRIPTION OF THE INVENTION 
System of the Invention 

Fig. 1A shows one embodiment of a system incorporating the present invention. In this 
embodiment, the system includes a central controller 110, configured to receive information 
from one or more email senders 124 or to send such information to one or more recipients 134 at 
user interface devices 120, 130. 

Central controller 110 preferably comprises a processor-based system that maintains 
databases and information relating to the preparation, transmission and receipt of multi-mail 
messages. A multi-mail message is an email message which has a multimedia component 
associated with the inputted message. The multimedia component may be in the form of audio 
content, image content (static or dynamic), graphic content or any other multimedia format. 
Further, the multi-mail message may be transmitted with a textual component in addition to one 
or more multimedia components, or, in an alternate embodiment, the textual component is 
replaced by one or more multimedia components. Although the multimedia component may be 
added to a message to facilitate conveying a user's emotion associated with a message, the 
component(s) may be utilized to simply more effectively transmit a sender's message. 

As described more fully below, a multi-mail message may be prepared in a number of 
different ways. In one embodiment, an email message may be prepared by typing in a textual 
message at a sender's user interface device 120 that is capable of receiving such text, and 
associating an accompanying audio and/or video component. In another embodiment, an email 
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message may be composed by telephone, or some other device that is capable of receiving an 
audio input. In one embodiment, user interface devices 120, 130 may be a computer comprising 
one or more central processing units (CPU), one or more data storage devices, a monitor, a 
keyboard and/or any other components that may allow a user to implement the commands of the 
software and hardware functions described herein. Further as Fig. 1 A illustrates, a sender user 
interface device may be a personal digital assistant (PDA) 120-1, telephone 120-2, or any type of 
wireless or land line device that is capable of transmitting textual, graphical or audio messages, 
such as facsimile machines, beepers, etc. Similarly, any devices that are capable of 
communicating an audio and/or video message (with or without transmitting textual data) are 
suitable for displaying email messages to a receiving party. 

Referring to Fig. IB, central controller 110 stores information received from senders or 
recipients in user/account database 352. Further, information concerning animations, images and 
audio components which may be associated to a specific message and/or a specific user is stored 
in databases 354, 356, 358. As described more fully below this information is used to compose, 
send and receive multi-mail messages which convey the sender's message in a manner that is 
not only textual. In some cases, the multi-mail message may have no textual component at all. 
The data stored in databases 354, 356, 358 effectuate the transmission of such message formats. 
The structure of certain embodiments of the central controller 1 10 is described below in 
connection with Fig. IB. 

Senders 124 include individual(s) wishing to compose and/or send a multi-mail message 
to recipient(s) 134. As described above, a multi-mail message is an electronic message that has 
at least one multimedia component associated with the message. In one embodiment, users 
(senders 124 and recipients 134) register, through user interface devices 120, 130, with the 
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service which performs the email processes as further described below. In one embodiment, the 
user interface device may be the user's computer or internet access device. In another 
embodiment, the user interface device may be a telephone, PDA, facsimile machine, etc. 
Senders 124 can input email message information in various ways. For instance, users may send 

5 such messages electronically by means of the internet, world wide web, intranet, or some other 
network. This may be accomplished by inputting a message through sender device 120 which is 
in communication with the network. In an alternate embodiment, the sender device may be in 
communication with some other type of network, such as the public switched telephone network 
(PSTN). Two ways in which a sender may input such information via the PSTN for receipt by 
lpM central controller 110 include (1) telephone answering services at central controller 1 10 that 

[ a ;t provide audio recording capabilities and, in some instances, that permit interaction between the 

rfi 

\ }| user and the system based on touch tone or voice input provided by the user; or (2) telephoning 

: :% 

live operators at central controller 1 10, to verbally provide information which is entered into the 
N system via operator terminals, 

W Figure IB illustrates one embodiment of the central controller 1 10 for a system according 

fss * to the present invention. As shown in Fig. IB, central controller 110 includes central processing 
unit (CPU) 240, random access memory (RAM) 220, read-only memory (ROM) 230, and large 
capacity storage device 250. CPU 240, preferably comprising a conventional microprocessor 
such as an Intel Pentium Processor, is electronically coupled to each of the central controller's 
20 110 other elements. 

CPU 240 executes program code stored in one or more of RAM 220, ROM 230 
and storage device 250 to carry out the functions and acts described in connection with central 
controller 110. CPU 240 preferably comprises at least one high-speed digital data processor 
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adequate to execute program modules for executing email transmissions by others 124, 134. 
These modules are described in connection with Figs. 2-17. CPU 240 interacts with RAM 220, 
ROM 230 and storage device 250 to execute stored program code according to conventional data 
processing techniques. 

5 User interface devices 120 and 130 comprise devices for allowing central 

controller 1 10 to communicate with multi-mail senders 124 and recipients 134. Communication 
between these user interface devices 120, 130 and the controller 1 10 is preferably electronic by 
means of the internet and preferably includes a conventional high speed modem employing 
known communication protocols capable of decrypting encrypted data received from the 

10^ interface user devices 120, 130. 

' s 4 

Large capacity storage device 250 contains transaction processor 260, 

"sir:? 

ijl user/account database 352, animation database 354, image database 356 and audio database 358. 
is Transaction processor 260 maintains, determines and accesses data stored in databases 352, 354, 
i s4 356, 358, and handles multi-mail message information for transmission between senders 124 and 
Ifjf recipients 134 as described in connection with Figs. 2-17. Transaction processor 260 may 
fa!if comprise a separate, conventional CPU/microprocessor, or a portion of the operating function of 
CPU 240. 

User/account database 352 contains data about senders and recipients that register 
with the multi-mail service described herein. The data is used to store subscriber information 
20 including name and email address(es), as well as information for accessing audio, video and 
animation files, where appropriate. Animation database 354 contains information relating to 
animations associated with a sender or sender's message. Animation information includes data 
relating to a dynamic graphic associated with a sender's message. Image database 356 contains 
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information relating to images associated with a sender or sender's message as well as data for 
controlling the audio and video aspects of a dynamic image. Image information may include any 
dynamic or static graphic associated with a sender's message. Audio database 358 contains 
information relating to audio data associated with a sender or sender's message. In one 
5 embodiment, the image may be a graphic of a user's, or someone else's, face. In an alternate 
embodiment, an image may be any other type of graphic. Audio information may include any 
voice(s) or sound(s) associated with a sender's message. As described more fully below, the 
animations, graphics and/or audio components associated with a sender's email message may 
include the likeness of the sender. In another embodiment, these multimedia components may 
lift! have little or no resemblance to the sender. The process of using data from user/account 
rj information database 352, animation database 354, image database 356 and audio database 358 
;ji to prepare, send and receive messages, as well as the other operations of the system described 
* with reference to Figs. 1A and IB, is represented in the flowcharts of Figs. 2-17, described in 
["7 detail below. Central controller 1 10 uses user/account information database 352, animation 
tl database 354, image database 356 and audio database 358 to execute multi-mail preparation and 
transmission processes- 
Login Process 

The login process is the preliminary process in which users (senders 124, recipients 134) 
20 provide information to effectuate access to the multi-mail system and serves at least the 

following purposes: (1) storing address information to enable proper routing of a multi-mail 
message; (2) transmitting messages (electronic, telephone, mail, etc.) by the multi-mail server to 
users such as new messages, return receipts, etc.; (3) effectuating login by comparing identifier 
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information entered by users during the login process to the stored personal identifier 
information provided at registration; and (4) enabling users to select a link so that message 
information and/or user information may be stored. 

Referring to Fig. 2, the login process begins at step 1000 where a user accesses the multi- 

5 mail program stored in central server 1 10 via user interface 120, 130. In one embodiment, this is 
accomplished by typing a universal resource locator (URL) associated with the multi-mail site. 
A determination is then made, in step 1 100, as to whether the user is an existing multi-mail user 
or a first time visitor. If the user has previously registered with the multi-mail site the user logs 
in by entering a username and associated password. This information is compared to the data 
10* previously stored in user/account database 352. Once the information is verified, the user may 

f ; further access the multi-mail site. 

Jl Once the login information is verified as being correct, the user may access several 

; s options including, but not limited to, managing graphics and audio files (step 1111), creating a 
new multi-mail (step 1112), accessing multi-mail via a free multi-mail account (step 1113) and 

l;S creating or editing user or multi-mail preferences (step 1114). Various aspects concerning 

managing graphics and audio files (step 1 1 1 1) are described below with reference to Figs. 4-11. 
Various aspects concerning creating a new multi-mail message (step 1112) are described below 
with reference to Figs. 12-17. Accessing multi-mail via a free email account (step 1113) allows 
a user to retrieve email sent to a personal email account at gabmail.tv, as well as email sent to 

20 them at other POP accounts. Finally, creating or editing preferences (step 1114) allows a user to 
select default settings associated with the user's account and/or messages. Such settings may 
include default audio, image and animation attributes, multi-mail routing information (i.e., 
priority information, forwarding information, etc.) and the like. The default settings may apply 
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to all or only a sub-group (such as those to a specific recipient 134) of multi-mail messages 
created by a user. 

If the user has not previously registered with the multi-mail site, a new username and 
password is established by the user and is stored in user/account database 352 (step 1 120). The 
5 user may then create preferences associated with the user's new account and/or individual 

messages (step 1114). Upon creating these preferences, the user may manage graphics and audio 
files (step 1 1 1 1), create a new multi-mail (step 1112), access multi-mail via a free multi-mail 
account (step 1113) and/or edit user or multi-mail preferences (step 1 1 14). 

1QB Telephone Access Process 

^ In addition to accessing the multi-mail system by computer, a user may access the system 

\n by telephone. (It should be noted that access to the multi-mail site is not limited only to 
: S computer and telephone access. These are merely representative samples of devices that may 
H access the system by either some graphical user interface (i.e., computer) or by audio interface 
lS-l (i.e., telephone). 

iss * In step 2000, a user accesses the multi-mail system through the PSTN by dialing a 

telephone, such as a toll free 800 number. Upon accessing the system, the user specifies, in step 
2100, whether they have an existing account. If an account does not already exist, the user may 
make such an indication by pressing a key on the telephone keypad (i.e., "*") or by speaking a 
20 predetermined word as requested by a multi-mail interactive voice response unit (step 2120). 

If, however, CPU 240 of central controller 110 determines that a user account already 
exists, the user's identification code and password are entered and compared to the data stored in 
user/account database 352. Upon verification that the user's login information is correct, the 
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user may access new (or old) received multi-mail messages (step 2111), compose a new multi- 
mail outgoing message (step 21 12) or change the user's personal settings (step 2113). Changing 
a personal setting may include recording a new audio component to be associated with multi- 
mail messages (step 21 13.1). This new audio component may be stored in audio database 358 in 
5 place of or in addition to any audio components that the user already has stored. Additional 
personal settings, such as choosing a default image (step 21 13.2), listening to other voice 
samples (21 13.3), or choosing background sounds (21 13.4), may be activated by the telephone 
system. Other options, such as recording samples of a laugh, a whisper, a shout, or the like are 
available. 

Q Graphics and Audio Management Process 

2 Referring to Fig. 4, various audio and graphics capabilities of the multi-mail system are 

: W illustrated at step 1 1 1 1 . For example, at step 1 1 1 1 . 1 , image or audio files may be uploaded. As 
El described above such files are stored in and may be accessed from any one or more of animation 
IIS database 354, image database 356 and audio database 358. These files may be .wav, .jpeg, 
Q .mpeg, or any other type of file containing audio or video data. In addition to storing such files, 
these files may be deleted from database 354, 356, 358 at step 1 1 1 1.2. 

Images that can be accessed by a user may also be manipulated by the user. In one 
embodiment, the manipulation may take place at the multi-mail website (i.e., central controller 
20 110) and may include enlarging or shrinking an image, reorienting the image, changing the 

colors of the image, tiling or cascading the image, etc. (see Fig. 7 and related description below 
for more detail). Further, at step 1 1 1.4, the graphics manipulation enables graphic add-ons 
which are described more fully below with reference to Fig. 8. 
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At step 1111.5, sound effects may be stored in and/or accessed from audio database 358. 
Sound effects may include changing the pitch, rate of speech, etc. of an existing message. 
Further, sound effects may include choosing from a library of common sounds including, but not 
limited to, the sound of breaking glass, wind blowing, door slamming, dog barking, etc. Another 
5 library of sound effects may be the voices of celebrities or other famous people. In other words, 
the sound effects may result from changes to a user's voice, another person's voice or any sound 
related or unrelated to a person's voice. Finally, at step 1 1 1 1 .6, a voice sample preference may 
be created or changed. Such preferences (1 1 1 1 .6) as well as the other graphic/audio 
management steps (1111.1-1111.5) are described more fully below with reference to Figs. 5-10. 

iq| 

Q Uploading an Image or Audio File 

^ An image or audio component may be associated with a multi-mail message by accessing 

™ from database 354, 356, 358 the file(s) which store such data. Referring to Fig. 5, a user who 
j a 4 wishes to upload an image or audio file (step 1111.1) may, in one embodiment, browse to or type 
15J i n the file location on a local computer (or some other user interface device 120 which may 
Q browse or access such files) to locate such data files (step 1111,11) or terminate the upload 

session. Once the appropriate file is located, the user may select and upload the data of such file 
(step 1111.12). The sound, image or graphic component may then be previewed at step 1111.13. 
If the component is not approved (step 1 1 1 1 . 14), the user may return to the browse mode (step 
20 1111.11). If, however, at step 1111.14, the component is approved, it is associated with the 

message and the user returns to component management menu (step 1111). Further, a keyword 
may be associated with the file so that the contents of the file may be accessed via telephone 
120-2. 
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Deleting an Image or Audio File 

Referring to Fig. 6, a user may choose to delete one or more audio, image or animation 
5 files (step 1 1 1 1.2). This is accomplished by first selecting an audio, image or animation from 
one or more available files stored in database 354, 356, 358 associated with the user's account 
(step 1 1 1 1.21). Once the file(s) are selected, the user is asked to confirm the deletion of the 
file(s) (step 1111 .22). If the selected file(s) are not confirmed for deletion, the user may choose 
to select other file(s) for deletion (or terminate the file deletion process). If, however, at step 
1(2 "1 1111 .22, the user confirms deleting the selected file(s), those file(s) are deleted from one or more 
=.p of databases 354, 356, 358 (1111 .23) and the user receives confirmation of such deletion 
U] (1111 .24). In one embodiment, the user is then returned to the component management menu 
f (step 1111). 

lS Manipulating an Image 

As described above, a user may wish to manipulate an image for association with a multi- 
mail message. Such manipulation begins at step 1111.3 of Fig.7. Image manipulation enables a 
user to easily change a stored image. Although a user may change a stored image in countless 
ways, some of the manipulation options are as follows: (1) include stock or own animation which 
20 could, for example, include a mini-movie in the background behind the talking head (step 
1 1 1 1 .3 1); (2) drag an image of a face into a goofy shape, (step 1111 ,32); (3) enlarge eyes or 
exaggerate other image features (step 1111.33); (4) provide color enhancement to an image or 
video (i.e., technicolor, grainy black and white, sepia, etc.) (step 1111.34); (5) enable image 
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movement, such as winking (step 111135); (6) change component background (i.e., street 
scenes, circus, ships, psychedelic, subtle, stripes, antique, etc.) (step 1111.36); (7) drag other 
objects into image or video (step 1111.37); and (8) move or alter body parts (step 1111.38). The 
manipulations listed above are merely representative examples of the effects that may be applied 
to a multi-mail image or video components and are not meant to be an exclusive list of 
manipulations available. After completing the image/video manipulation (step 1 1 1 1.3), the user 
may return to the component management menu (step 1111). 

Graphic Add-On Process 

As shown in Fig. 8, a user may also wish to apply graphic add-ons to a multi-mail 
message (step 1111 .4). These add-ons enable a user to introduce additional images into a 
graphic component. The additional image may already be stored in image database 356, may be 
accessed by some other database storing such images or may be scanned into the user's local 
user interface device 120. Although a user may access countless graphic add-ons, some of the 
add-ons that may be applied to a multi-mail message are as follows: (1) applying various body 
types, including different sizes and proportions, color (skin, hair, etc.), features, etc. (step 
1 1 1 L41); (2) including different types of clothing and accessories to images of persons (step 
1111 .42); (3) applying different tints to the graphic (step 1111 .43); and (4) applying different 
surroundings and wallpapers to the graphic (step 1111 .44). The graphic add-ons listed above are 
merely representative examples of the add-ons that may be applied to a multi-mail image or 
video component and are not meant to be an exclusive list of the available graphic add-ons. 
After completing the graphic add-ons process (step 1111 .4), the user may return to the 
component management menu (step 1111). 
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Sound Effect Process 

As shown in Fig. 9, a user may also wish to apply sound effects to a multi-mail message 
(step 1111 .4). These sound effects enable a user to include either speech or non-speech sound 
that is associated with a multi-mail message. In one embodiment, the sound effect may simply 
be the user stating the textual contents of the multi-mail message using his or her normal voice. 
In another embodiment, the sound effect may be the user's speech attenuated to convey an 
emotion associated with the message. In yet another embodiment, the sound may be the speech 

; ;:;:3, 

3 of a person other than the user. Finally, in another embodiment, the sound effect may be some 
Q noise that is not speech. Some examples of the sound effects that may be applied to a multi-mail 

message are as follows: (1) applying joke or gag sounds (step 1111.51); (2) including 
y background music (step 1 1 1 1.52); and (3) choosing from generic voice samples (step 1111 .53). 
U The sound effects listed above are merely representative examples of the sounds that may be 
Q applied to a multi-mail image or video component. The sound effects may already be stored in 
Q audio database 358, may be accessed by some other database storing such sounds or may be 

recorded into local storage of a local user interface device 120 that is capable of receiving sound 

data. After completing the sound effects process (step 1111.5), the user may return to the 

component management menu (step 1111). 

> 

Establishing or Changing Voice Sample Preferences 

Referring to Fig. 10, voice sample preferences may be established or changed (step 
1111 .6). A voice sample is any voice that a user may store and/or access for association with a 
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multi-mail message. As described more fully above, the voice may be that of a user, a variation 
of a user's voice, or someone else's voice. 

In one embodiment, a voice sample may be stored in audio database 358 by telephone 
120-2 or any other device having an audio input and output in communication with central 
controller 110. At step 11 11.61, a user initiates the voice sample call in process by accessing a 
telephone number (such as a toll free telephone number) which is associated with the multi-mail 
message service. The user logs into the system by providing a username and password (step 

1111.611) . The user then navigates to a voice sample recording section of the system (step 

1 1 1 1 .612) . In one embodiment, this is accomplished by pressing the appropriate button on a 
keypad in response to the system requests. In an alternate embodiment, an interactive voice 
response unit may be used. At step 1111.613, the voice sample is recorded. Recording devices 
may be part of the telephone system. The user will read a statement into the phone that will 
provide a broad enough spectrum of sounds to allow the automated synthesis of any text the user 
may type in the future. The user may then review the recorded sample at step 1 1 1 1 .614. If the 
user does not approve of the recorded sample (step 1 1 1 1 .61 5), the process returns to step 

1 1 1 1.613 where the voice sample may be re-recorded (or the process may be terminated). If, 
however, the recording is approved, the user may confirm such approval (step 1111.616). The 
system saves and then parses the voice sample into phonemes (step 1 1 1 1 .617) and organizes, 
sorts and stores the phonemes (step 1 1 1 1 .61 8). More specifically, the system will (1) analyze 
sample for breaks (silence) and divide it into words at breaks; (2) divide words into syllables; (3) 
divide syllables into component vowel and consonant sounds, or phonemes; (4) identify each in 
the set of phonemes thus derived from voice sample by comparing to a standard set of spoken 
phonemes; and (5) eliminate duplicates, enumerate any missing phonemes, and organize 
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collection for future use. At this point, a person's voice is stored in such manner that the multi- 
mail system may transmit a voice-based message to a recipient 134. In another embodiment, an 
audio component may be associated with a textual message by converting the textual message to 
an audio one. This may be accomplished by utilizing a text to speech converter which is known 
5 in the art. 

In addition to calling in a voice sample (step 1111.61), a user may choose a voice sample 
from a generic voice sample stored in, for example, audio database 358 (step 1111 .62). Further, 
an audio file may be uploaded as described above (step 1111.1). 

1 Qy Creating a New Message 

Q Referring to Fig. 1 1, a user may create a new multi-mail message at step 1 1 12. This is 

^ accomplished by (1) creating an audio portion (step 1 1 12.1), (2) selecting a preexisting image or 

. ft 

^ uploading a new image (step 1 1 12.2), (3) processing the image (step 1 1 12.3), and (4) 
M synthesizing the multi-mail message (step 1 1 12.4). Steps 1 1 12.1-1 1 12.4 are described in greater 
1^3 detail below with reference to Figs. 12-17. It should be noted that although the multi-mail system 
Q supports all types of multimedia components (i.e., audio, image, animation, etc.), there is no 
requirement that more than one component be selected by the user. For example, the user may 
wish to add only an audio component, in the form of the user's voice, to an otherwise standard 
text email. Alternatively, the user may want to show a picture of an object in association with 
20 the message. 

Once a multi-mail message has been composed with the appropriate associated audio, 
image and/or animation components, the user may preview the multi-mail message at step 
1 1 12.5. If the user does not approve the message (step 1 1 12.6), the multi-mail message 
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composition process may be terminated or the user may return to step 1 1 12.1 where the message 
may be edited or created anew in part or in toto. If, however, at step 1 1 12.6, the user approves 
the multi-mail message, the user may choose to transmit the multi-mail message (step 1 1 12.7). 
In one embodiment, the message is transmitted as soon as the multi-mail message is approved. 
5 In an alternate embodiment, the user may choose an option to have the message sent either 
automatically or manually at some future time. In any event, when the message is transmitted, 
the user may select an option to receive a confirmation (step 1 1 12.8) when the transmitted 
message has been sent by the sender 124 and/or received by the recipient 134. 

1 GO Creating an Audio Portion of Message 

l % As described above, a multi-mail message may include an audio component to more 

■HI 

;1i effectively convey a message. Creating the audio component of a message begins at step 1112.1 
r of Fig. 12. Entering the audio component of a multi-mail message into the mail system may be 
H accomplished in at least two ways. In one embodiment, a sender 124 may enter text via user 
if) interface device 120 (step 1 1 12.1 1). CPU 240 of central controller 110 may then synthesize the 
J typed text into audio data which is stored in audio database 358 (step 1112.111). This is 

accomplished by using a text to speech converter in conjunction with the parsed voice sample 
stored in audio database 358 that is selected by sender 124. 

In an alternate embodiment, the sender 124 may submit a spoken message by telephone 
20 120-2 or some other device which has an audio input and output capability and is in 

communication with central controller 1 10 (step 1 1 12.12). Next, central controller translates the 
sound to text by accessing a speech to text converter. The conversion is performed and saved at 
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step 1112.121. Sound quality processing is performed at step 1 1 12. 122 and the audio component 
creations process is complete (step 1 1 12.1). 

Image Processing 

5 As described above, a multi-mail message may include an image component to more 

effectively convey a message. Creating the image component of a message begins at step 1112.3 
of Fig. 13. The image processing module enables a user to store his or her own likeness or that of 
another (using a digital camera, developing a standard photograph digitally onto a CD, scanning 
an image, or using some other digital input device) to more effectively convey a message. 
VOIl Further, the module enables the user to combine the audio component with a dynamic image 
^ component, to enable a recipient 134 to receive a message wherein a person's image and voice is 

t conveying a message. 
;g * At step 1 1 12.41, the user indicates to CPU 240 of central controller 1 10 whether the 

i«* image is a face (step 1 1 12.41). If the image is a face, the eyes and mouth are located (at step 
£S 1 1 12.41 1). An image recognition application that looks for almond shapes (eyes), oval 

diamonds (mouths), or ovals (faces) may be either custom written or existent and known to those 
skilled in the art. Further, mouth shape frames are generated for each corresponding message 
vowel and consonant (step 1 1 12.41 1) and eye blink frames are generated as well (step 
1 1 12.41 12). The system includes custom written code that utilizes an algorithm or set of 
20 algorithms to make minute changes to an image, save in a new file, make more minute changes, 
save again in a new file, and so on until the collection of images together form the frames of 
animation to look like a speaking face. The images would then be compiled in order forming an 
animated sequence. If the image is not a face (or even if it is) the user may use a custom-written 
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web-based application to denote regions of their image they would like to animate, and then 
select from a pre-determined collection of types of animation (blinking eyes, moving mouth, 
nodding head, or limited set of other non-facial movements) to apply to those regions of the 
image. 

5 

EMail Creation Process 

Referring to Fig. 14, the multi-mail message is created at step 1 1 12.4. In one 
embodiment, the text is first entered into the multi-mail message file (step 1 1 12.41). Next, an 
associated sound clip is parsed into phonemes (step 1 1 12.42) if the audio component was 
l l previously entered by telephone or a similar audio device. If, however, the message was 
'3 originally typed in (i.e., textual), the phoneme information is recalled from the parsed typed text 
/•~ which is stored in image and audio databases 354, 358 (step 1 1 12.41 a). 

Next, at step 1 1 12.43, a phoneme/time sequence is composed by CPU 240 . Further, a 
U matching mouth frame/time sequence (step 1 1 12.44) and a non-matching eye blink frame 
13 sequence (step 1 1 12.45) are composed. The sequence of image frames is composed to match the 
9 sequence of phonemes, in order and in individual frame duration. The image sequence and 
sound information are then combined into a synchronized clip which is stored in animation 
database (step 1 1 12.46). Finally, the animation clip is embedded into hypertext markup 
language (HTML) in a multi-mail message for transmission to a recipient's user interface device 
20 130. 

Text to Audio Synthesis Process 
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Referring to Fig. 15, the text to audio synthesis process is described. A sender most 
frequently enters an email message by typing in a text message which is displayed as a graphical 
user interface on the sender's user interface device 120 and then sent to a recipient's 134 user 
interface device 130. Because multi-mail messages have a multimedia component associated 

5 with the text message, the central controller converts the text message to an audio/image display 
to accompany and/or replace the text display. The synthesis enables such display to look and 
sound life-like. The text to audio synthesis begins at step 1112.111. First, central controller 1 1 0 
parses the textual message into sentences. This may be accomplished by recognizing 
punctuation provided by the user or some other algorithm known in the art. Next, at step 
1(23 1 1 12.1 12, voice modulation controls are assigned to sentence end. This enables appropriate 

Q inflection and pausing to be accomplished as sentences end. 

;tj At step 1 1 12. 1 1 1 3, the text is then parsed for voice modulation controls, or custom codes 

to indicate a variety of voice inflections or adjustments to be made on that sound. The sentence 
U type (i.e., question, compound sentence, etc.), as well as emphasis commands provided by the 
1® sender 124, may dictate how the text is parsed. Next, phoneme and modulation information is 
Q sequenced and translated into a sound sequence (steps 1 1 12. 1 1 14 and 1 1 12. 1 1 1 5) and the voice 
is modulated pursuant to the modulation controls or commands (step 1 1 12.1 1 16). Using sound 
effect plug-in tools known to those skilled in the art, such as Mark of the Unicorn's Spectral 
Effects, the voice can be modulated in terms of pitch (higher or lower), formants (phoneme 
20 energy), or speed (time expansion or compression), to indicate a range and intensity of emotion. 
Finally, the sound quality is processed, at step 1 1 12.21, and the associated multi-mail message 
audio file is compiled (step 1112.1117). The user may then send the message with the associated 
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audio file; the associated text may be optionally transmitted. Further, as described above, an 
associated image file may also be associated with the message and transmitted. 

Receiving Spoken Message by Telephone 

Referring to Fig. 16, the process of receiving an audio message via telephone 120-2 or 
some other device having audio input and output capabilities is described. At step 1 1 12.12, a 
sender 124 submits a spoken message by telephone 120-2. In one embodiment, the voice sample 
may be stored in audio database 358. 

The user first accesses the system by calling a telephone number (such as a toll free 
telephone number) which is associated with the multi-mail message service. The user logs into 
the system by providing a username and password (step 1 1 12. 121). The user then navigates to a 
voice sample recording section of the system (step 1 1 12.1 12). In one embodiment, this is 
accomplished by pressing the appropriate button on a keypad in response to the system requests. 
In an alternate embodiment, an interactive voice response unit may be used. At step 1 1 12.123, 
the voice sample is recorded. The multi-mail message service system contains software whose 
function it is to digitize (i.e., record) sound. At the appropriate signal from the multi-mail 
message service, this software is invoked to record the user's voice sample. The user may then 
review the recorded sample at step 1 1 12.124. If the user does not approve of the recorded 
sample (step 1 1 12.6), the process returns to step 1111 .123 where the voice sample may be re- 
recorded (or the process may be terminated). If, however, the recording is approved, the user 
may confirm such approval (step 1111 .126). The system then performs the necessary recorded 
message processing (step 1 1 12.127) as described in detail above with reference to steps 
1111.617 and 1111.618 of Fig. 10. 
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Sound Quality Processing 

Fig. 17 illustrates the sound quality processing of the system shown in Figs. 1 A and IB. 
Sound quality processing is comprised of the different modules that central controller 1 10 
performs to the audio component of a multi-mail message to ensure that the sound has realistic 
qualities. At step 1 1 12.21 1, the equalization process is performed to adjust individual sound 
wave frequencies. Next, at step 1 1 12.212, central controller 1 10 performs the compression 
process, smoothing out sound wave peaks and compressing audio frequencies. Next, at step 
1 1 12.213, noise gating is performed, which Anther improves the sound quality by only 
permitting the audio component to pass if it reaches a certain user defined threshold, and an 
audio limiting process is performed by central controller 1 10, which limits the audio signal at a 
user defined level. These audio processes, eliminating background noise and other distortions 
and improving overall audio quality using methods known to those skill in the art, provide a 
high quality audio component to be communicated to a recipient 134. 

It will be apparent to those skilled in the art that various modifications and variations can 
be made in the system and processes of the present invention without departing from the spirit or 
scope of the invention. Thus, it is intended that the present invention cover the modifications 
and variations of this invention provided they come within the scope of the appended claims and 
their equivalents. In this context, equivalents means each and every implementation for carrying 
out the functions recited in the claims, even if not explicitly described herein. 
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CLAIMS 

What is claimed is: 

1 . A method for preparing a multi-mail message for transmission over a network, 
comprising: 

5 receiving data comprising textual content of said message; 

creating one or more multimedia components associated with said message, 
wherein said multimedia component represents a likeness of a sender; and 

synthesizing said multimedia components with said textual content. 



103 2. The method according to claim 1, wherein the multimedia component comprises 

audio information. 



ife -* 3. The method according to claim 1, wherein the multimedia component comprises 

[ I image information. 

Q 4. The method according to claim 3, wherein the image information may be static or 

dynamic. 

5. The method according to claim 2, wherein the audio component comprises voice 
20 data that enables the generation of sounds similar to the user's voice speaking the words of the 
textual content of the message. 
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6. The method according to claim 2, wherein the audio component comprises voice 
data that enables the generation of sounds similar to a generic voice sample. 

7. The method according to claim 2 ? wherein the audio component comprises voice 
5 data that enables the generation of any stored sound. 

8. The method according to claim 2 wherein the synthesis of the multimedia 
component containing audio information further comprises: 

parsing the audio information into sentences and for voice modulation controls; 
lQa assigning voice modulation to audio information; 

Q sequencing phoneme and modulation information; and 

|^ translating said phoneme sequence into a sound component sequence. 

9. The method according to claim 1, wherein the synthesis of the image multimedia 
1 £h component further comprises: 

Q identifying speech movement image feature; and 

generating frames representing movement of said image features. 

1 0. The method of claim 1 , wherein the multimedia component comprises audio 
20 information and image information. 

11. The method according to claim 10, wherein the synthesis of the audio and image 
multimedia components further comprises: 
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composing a phoneme sequence; 

composing a mouth frame time sequence which matches the phoneme time 

sequence; 

composing speech movement image frame sequence; and 
combining the image and phoneme sequences. 

12. The method according to claim 10, further comprising: 

varying one or more of said components to convey one or more senses of said 
message content. 

13. The method according to claim 12, wherein the senses of said message content 
correspond to one or more sender emotions associated with said message. 

14. The method according to claim 13, wherein the sender emotions are conveyed by 
manipulating one or more said image components. 

15. The method according to claim 13, wherein the sender emotions are conveyed by 
manipulating one or more said audio components. 

16. The method for preparing a multi-mail message for transmission over a network, 
comprising: 

receiving audio information relating to content of said message; 
translating said audio information to text information; and 
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combining said audio information and text information in preparation of said 

transmission. 

17. The method according to claim 16, wherein the audio information is similar to 
5 sound generated by the user's voice. 

18. The method according to claim 16, further comprising: 

generating sound similar to a generic voice sample based on the received audio 

information. 

io3 

•3 19. The method according to claim 16, further comprising: 

generating one or more stored sounds based on the received audio information. 



20. The method according to claim 1 9, further comprising: 
1 5 Q creating one or more multimedia components associated with said message 



'Sit';? 



content. 



21 . The method according to claim 20, wherein the multimedia component comprises 
image information. 

20 

22. The method according to claim 21 , wherein the image information may be static 
or dynamic. 
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5 



23. The method according to claim 21, wherein the synthesis of the image multimedia 
component further comprises: 

identifying speech movement image feature; and 

generating frames representing movement of said image features. 

24. The method of claim 16, wherein the multimedia component comprises audio 
information and image information. 



lO^J 25. The method according to claim 24, wherein the synthesis of the audio and image 

Q multimedia components further comprises: 

composing a phoneme sequence; 
^ composing a mouth frame time sequence which matches the phoneme time 

[]: sequence; 

15 p composing speech movement image frame sequence; and 

combining the image and phoneme sequences. 



i 5? 



26. The method according to claim 24, further comprising: 

varying one or more of said components to convey one or more senses of said 
20 message content. 

27. The method according to claim 24, wherein the senses of said message content 
correspond to one or more sender emotions associated with said message. 
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28. The method according to claim 27, wherein the sender emotions are conveyed by 
manipulating one or more said image components. 

29. The method according to claim 27, wherein the sender emotions are conveyed by 
manipulating one or more said audio information. 

30. A system for preparing a multi-mail message for transmission over a network, 
comprising: 

means for receiving data comprising textual content of said message; 
means for creating one or more multimedia components associated with said 
message, wherein said multimedia component represents a likeness of a sender; and 

means for synthesizing said multimedia components with said textual content. 

3 1 . The system according to claim 30, wherein the multimedia component comprises 
audio information. 

32. The system according to claim 30, wherein the multimedia component comprises 
image information. 

33. The system according to claim 32, wherein the image information may be static or 
dynamic. 
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34. The system according to claim 3 1, wherein the audio component comprises voice 
data that enables the generation of sounds similar to the user's voice. 



35. The system according to claim 3 1 , wherein the audio component comprises voice 
data that enables the generation of sounds similar to a generic voice sample. 

36. The system according to claim 3 1 , wherein the audio component comprises voice 
data that enables the generation of any stored sound. 



^ 37 « The system according to claim 3 1 wherein the synthesis of the multimedia 

^ component containing audio information further comprises: 

m means for parsing the audio information into sentences and for voice modulation 



P controls; 



means for assigning voice modulation to audio information; 

means for sequencing phoneme and modulation information; and 

means for translating said phoneme sequence into a sound component sequence. 



38. The system according to claim 39, wherein the synthesis of the image multimedia 
component further comprises: 

means for identifying speech movement image feature; and 

means for generating frames representing movement of said image features. 
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39. The system of claim 30 5 wherein the multimedia component comprises audio 
information and image information. 

40. The system according to claim 39, wherein the synthesis of the audio and image 
5 multimedia components further comprises: 

means for composing a phoneme sequence; 

means for composing a mouth frame time sequence which matches the phoneme 

time sequence; 

means for composing speech movement image frame sequence; and 
1 Gjj means for combining the image and phoneme sequences. 

:^ 41 . The system according to claim 39, further comprising: 

means for varying one or more of said components to convey one or more senses 
] es i of said message content. 
15:3 

C- 42. The system according to claim 41, wherein the senses of said message content 

correspond to one or more sender emotions associated with said message. 

43. The system according to claim 42, wherein the sender emotions are conveyed by 
20 manipulating one or more said image components. 

44. The system according to claim 42, wherein the sender emotions are conveyed by 
manipulating one or more said audio components. 
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45. The system for preparing a multi-mail message for transmission over a network, 
comprising: 

means for receiving audio information relating to content of said message; 
5 means for translating said audio information to text information; and 

means for combining said audio information and text information in preparation 
of said transmission. 

46. The system according to claim 45, wherein the audio information is similar to 
1 Qy sound generated by the user's voice. 



% 47. The system according to claim 45, further comprising: 

'!* means for generating sound similar to a generic voice sample based on the 

!S | received audio information. 



Q 48. The system according to claim 45, further comprising: 

means for generating one or more stored sounds based on the received audio 

information. 

20 49. The system according to claim 48, further comprising: 

means for creating one or more multimedia components associated with said 
message content. 
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50. The system according to claim 49, wherein the multimedia component comprises 
image information. 

51. The system according to claim 50, wherein the image information may be static or 
dynamic. 

52. The system according to claim 50, wherein the synthesis of the image multimedia 
component further comprises: 

means for identifying speech movement image feature; and 

means for generating frames representing movement of said image features. 

53. The system of claim 45, wherein the multimedia component comprises audio 
information and image information. 



54. The system according to claim 53, wherein the synthesis of the audio and image 
multimedia components further comprises: 

means for composing a phoneme sequence; 

means for composing a mouth frame time sequence which matches the phoneme 

time sequence; 

means for composing speech movement image frame sequence; and 
means for combining the image and phoneme sequences. 



55. The system according to claim 53, further comprising: 
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means for varying one or more of said components to convey one or more senses 
of said message content. 

56. The system according to claim 53 , wherein the senses of said message content 
correspond to one or more sender emotions associated with said message. 

57. The system according to claim 56, wherein the sender emotions are conveyed by 
manipulating one or more said image components. 



lOg 58. The system according to claim 56, wherein the sender emotions are conveyed by 



*4 manipulating one or more said audio information. 



59. A system for preparing a multi-mail message for transmission over a network, 
comprising: 

a database for receiving data comprising textual content of said message; and 
a central processing unit configured for creating one or more multimedia 
components associated with said message, wherein said multimedia component represents a 
likeness of a sender, and for synthesizing said multimedia components with said textual content. 

60. The system according to claim 59, wherein the multimedia component comprises 
audio information. 
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61 . The system according to claim 59, wherein the multimedia component comprises 
image information. 

62. The system according to claim 61, wherein the image information may be static or 
dynamic. 

63. The system according to claim 60, wherein the audio component comprises voice 
data that enables the generation of sounds similar to the user's voice. 

64. The system according to claim 60, wherein the audio component comprises voice 
data that enables the generation of sounds similar to a generic voice sample. 

65. The system according to claim 60, wherein the audio component comprises voice 
data that enables the generation of any stored sound. 

66. The system according to claim 60 wherein the central processing unit is further 
configured for parsing the audio information into sentences and for voice modulation controls, 
for assigning voice modulation to audio information, for sequencing phoneme and modulation 
information, and for translating said phoneme sequence into a sound component sequence. 

67. The system according to claim 60, wherein the central processing unit is further 
configured for identifying speech movement image feature, and for generating frames 
representing movement of said image features. 
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68. The system of claim 59, wherein the multimedia component comprises audio 
information and image information. 

69. The system according to claim 68, wherein the central processing unit is further 
configured for composing a phoneme sequence, for composing a mouth frame time sequence 
which matches the phoneme time sequence, for composing speech movement image frame 
sequence, and for combining the image and phoneme sequences. 



ICR 70. The system according to claim 68, wherein the central processing unit is further 

Q configured for varying one or more of said components to convey one or more senses of said 
message content. 



; si 



Li 7 1 • The system according to claim 70, wherein the senses of said message content 

1 5 Q corr espond to one or more sender emotions associated with said message. 

72. The system according to claim 71, wherein the central processing unit is further 
configured for manipulating one or more said image components to convey said sender 
emotions. 



20 



73. The system according to claim 71, wherein the central processing unit is further 
configured for manipulating one or more said audio components to convey said sender emotions. 
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74. The system for preparing a multi-mail message for transmission over a network, 
comprising: 

a database for receiving audio information relating to content of said message; 

and 

a central processing unit for translating said audio information to text information, 
and for combining said audio information and text information in preparation of said 
transmission. 

75. The system according to claim 74, wherein the audio information is similar to 
sound generated by the user's voice. 

76. The system according to claim 74, wherein the central processing unit is further 
configured for generating sound similar to a generic voice sample based on the received audio 
information. 

77. The system according to claim 74, wherein the central processing unit is further 
configured for generating one or more stored sounds based on the received audio information. 

78. The system according to claim 77, wherein the central processing unit is further 
configured for creating one or more multimedia components associated with said message 
content. 
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79. The system according to claim 78, wherein the multimedia component comprises 
image information. 

80. The system according to claim 79, wherein the image information may be static or 
dynamic. 

8 1 . The system according to claim 79, wherein the central processing unit is further 
configured for identifying speech movement image feature, and for generating frames 
representing movement of said image features. 

82. The system of claim 74, wherein the multimedia component comprises audio 
information and image information. 

83. The system according to claim 82, wherein the central processing unit is further 
configured for composing a phoneme sequence, for composing a mouth frame time sequence 
which matches the phoneme time sequence, for composing speech movement image frame 
sequence, and for combining the image and phoneme sequences. 

84. The system according to claim 82, wherein the central processing unit is further 
configured for varying one or more of said components to convey one or more senses of said 
message content. 
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85. The system according to claim 82, wherein the central processing unit is further 
configured for varying one or more of said components to convey one or more senses of said 
message content. 

86. The system according to claim 85, wherein the senses of said message content 
correspond to one or more sender emotions associated with said message. 

87. The system according to claim 85, wherein the central processing unit is further 
configured for manipulating one or more said image components to convey said sender 
emotions. 

88. Computer executable software code stored on a computer readable medium, the 
code for preparing a multi-mail message for transmission over a network, comprising: 

code to receive data comprising textual content of said message; 
code to create one or more multimedia components associated with said message, 
wherein said multimedia component represents a likeness of a sender; and 

code to synthesize said multimedia components with said textual content. 

89. Computer executable software code stored on a computer readable medium, the 
code for preparing a multi-mail message for transmission over a network, comprising: 

code to receive audio information relating to content of said message; 
code to translate said audio information to text information; and 
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code to combine said audio information and text information in preparation of 
said transmission. 

90. A programmed computer for preparing a multi-mail message for transmission 
over a network, comprising: 

a memory having at least one region for storing computer executable program 

code; and 

a processor for executing the program code stored in memory, wherein the 
program code includes: 

code to receive data comprising textual content of said message; 
code to create one or more multimedia components associated with said 
message, wherein said multimedia component represents a likeness of a sender; and 

code to synthesize said multimedia components with said textual content. 

91. A programmed computer for preparing a multi-mail message for transmission 
over a network, comprising: 

a memory having at least one region for storing computer executable program 

code; and 

a processor for executing the program code stored in memory, wherein the 
program code includes: 

code to receive audio information relating to content of said message; 
code to translate said audio information to text information; and 
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code to combine said audio information and text information in 
preparation of said transmission. 



92. A computer readable medium having computer executable software code stored 
thereon, the code for preparing a multi-mail message for transmission over a network, 
comprising: 

code to receive data comprising textual content of said message; 
code to create one or more multimedia components associated with said message, 
wherein said multimedia component represents a likeness of a sender; and 

code to synthesize said multimedia components with said textual content. 

93 . A computer readable medium having computer executable software code stored 
thereon, the code for preparing a multi-mail message for transmission over a network, 
comprising: 

code to receive audio information relating to content of said message; 
code to translate said audio information to text information; and 
code to combine said audio information and text information in preparation of 
said transmission. 
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ABSTRACT 

A multi-mail system and method is disclosed in which a sender may convey and a 
recipient can realize emotional aspects associated with substantive content of a multi-mail 
message by receiving a message that is more than textual in nature. Voice recognition 
technology and programmatic relation of sound and graphics may be used to produce a talking 
image. In one embodiment, the image may include the user's own visual and/or audio likeness. 
In an alternate embodiment, the image may comprise any available visual and/or audio display 
selected by the user. The multi-mail message may be inputted by a user in a text format and 
transposed into a format including the selected image and/or voice. In an alternate embodiment, 
a spoken message may be converted into a format including the selected image and/or voice. 
The formatted messages are then stored and/or transmitted via an email system or some other 
electronic network. 
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COMBINED DECLARATION AND POWER OF ATTORNEY FOR 
ORIGINAL, DESIGN, NATIONAL STAGE OF PCT, SUPPLEMENTAL 
DIVISIONAL, CONTINUATION OR CONTINUATION-IN-PART APPLICATION 

As a below name inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name, 

I believe I am the original, first and sole inventor (if only one name is listed below) or an original, first and joint 
inventor (if plural names are listed below) of the subject matter which is claimed and for which a patent is sought on 
the invention entitled: 

ELECTRONIC MAIL METHOD AND SYSTEM USING ASSOCIATED AUDIO AND VISUAL TECHNIQUES 
the specification of which 

a. [x] is attached hereto 

b. [ ] was filed on as application Serial No, and was amended on 

. (if applicable). 

PCT FILED APPLICATION ENTERING NATIONAL STAGE 

c. [ ] was described and claimed in International Application No. filed on and 

as amended on . (if any). 

I hereby state that I have reviewed and understand the contents of the above-identified specification, including the 
claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose information which is material to the patentability as defined in Title 37, Code of 
Federal Regulations, § 1.56. 

I hereby specify the following as the correspondence address to which all communications about this application are 
to be directed: 

SEND CORRESPONDENCE TO: MORGAN & FINNEGAN, L.L.P 

345 Park Avenue 
New York, N.Y. 10154 

DIRECT TELEPHONE CALLS TO: Mark J". Schildkraut 
(212) 758-4800 

[ ] I hereby claim foreign priority benefits under Title 35, United States Code § 1 19(a)-(d) or under 
§ 365(b) of any foreign appHcation(s) for patent or inventor's certificate or under § 365(a) of any PCT international 
application(s) designating at least one country other than the U.S. listed below and also have identified below such 
foreign application(s) for patent or inventor's certificate or such PCT international application(s) filed by me on the 
same subject matter having a filing date within twelve (12) months before that of the application on which priority is 
claimed: 

[ ] The attached 35U.S.C.§ 119 claim for priority for the application(s) listed below forms a part of this 
declaration. 
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Application Date of filing Date of Issue Priority 

Countrv/PCT Number f day, month, vr) (day, month, yr) Claimed 



[1 YES \ 1 NO 

r i yes r i no 

r T YES [ 1NO 



[ ] I hereby claim the benefit under 35 U.S. C. § 1 19(e) of any U.S. provisional applications) listed below. 
Provisional Application No. Date of Filing (day, month, yr) 



ADDITIONAL STATEMENTS FOR DIVISIONAL, CONTINUATION OR CONTINUATION-IN-PART 
OR PCT INTERNATIONAL APPLICATIONS (DESIGNATING THE U.S.) 

I hereby claim the benefit tinder Title 35, United States Code § 120 of any United States application(s) or under 
§ 365(c) of any PCT international application^) designating the U.S. listed below. 



US/PCT Application Serial No. Filing Date Status (patented, pending, abandoned)/ 

U.S. application no. assigned (For PCT) 



US/PCT Application Serial No. Filing Date Status (patented, pending, abandoned)/ 

U.S. application no. assigned (For PCT) 

[ ] In this continuation-in-part application, insofar as the subject matter of any of the claims of this 
application is not disclosed in the above listed prior United States or PCT international application(s) in the manner 
provided by the first paragraph of Title 35, United States Code, § 1 12, 1 acknowledge the duty to disclose material 
information as defined in Title 37, Code of Federal Regulations, § 1.56(a) which occurred between the filing date of 
the prior application(s) and the national or PCT international filing date of this application. 

I hereby declare that all statements made herein of my own knowledge are true and that all statements made on 
information and belief are believed to be true; and further that these statements were made with the knowledge that 
willful false statements and the like so made are punishable by fine or Imprisonment, or both, under Section 1001 of 
Title 18 of the United States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 
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I hereby appoint the following attorneys and/or agents with full power of substitution and revocation, to prosecute 
this application, to receive the patent, and to transact all business in the Patent and Trademark Office connected 
therewith: John A. Diaz (Reg. No. 19,550), John C. Vassil (Reg. No. 19,098), Alfred P. Ewert (Reg. No. 19,887) 
David H. Pfeffer (Reg. No. 19,825), Harry C. Marcus (Reg. No. 22,390), Robert E. Paulson (Reg. No. 21,046) 
Stephen R. Smith (Reg. No. 22,615), Kurt E. Richter (Reg. No. 24,052), J. Robert Dailey (Reg. No. 27,434), Eugene 
Moroz (Reg. No. 25,237), John F. Sweeney (Reg. No. 27,471), Arnold I. Rady (Reg. No. 26,601), Christopher A. 
Hughes (Reg. No. 26,914), William S. Feiler (Reg. No. 26,728), Joseph A. Calvaruso (Reg. No. 28,287), James W. 
Gould (Reg. No. 28,859), Richard C. Komson (Reg. No. 27,913), Israel Blum (Reg. No. 26,710), Bartholomew 
Verdirame (Reg. No. 28,483), Maria C.H. Lin (reg. No. 29,323), Joseph A. DeGirolamo (Reg. No. 28,595), Michael 
P. Dougherty (Reg. No. 32,730), Seth J. Atlas (Reg. No. 32,454), Andrew M. Riddles (Reg. No. 31,657), Bruce D. 
DeRenzi (Reg. No. 33,676), Michael M. Murray (Reg. No. 32,537), Mark J. Abate (Reg. No. 32,527), John T 
Gallagher (Reg. No. 35,5 16), Steven F. Meyer (Reg. No. 35,613), Kenneth H. Sonnenfeld (Reg. No. 33,285), Tony 
V. Pezzano (Reg. No. 38,271), Andrea L. Wayda (Reg. No. 43,979) and Walter G. Hanchuk Reg. No. (35,179) of 
Morgan & Finnegan, L.L.P. whose address is: 345 Park Avenue, New York, New York, 10154; and Michael S 
Marcus (Reg. No. 31,727) and John E. Hoel (Reg. No. 26,279) of Morgan & Finnegan, L.L.P., whose address is 
1775 Eye Street, Suite 400, Washington, D.C. 20006. 

[ ] I hereby authorize the U.S. attorneys and/or agents named hereinabove to accept and follow instructions 
from 

. as to any action to be taken in the U.S. Patent and Trademark Office 



regarding this application without direct communication between the U.S. attorneys and/or agents and me. 
In the event of a change in the person(s) from whom instructions may be taken I will so notify the U.S. 
attorneys and/or agents hereinabove. 




Full name of sole or first inventor^ / Adam Rpth 
Inventor's signature* 
Residence 



. — * j / c. s uaic 



Citizenship ^/_5s 



Post Office Address Same as residence address above 



Inventor's signature* 




Full name of second joint inventor, if any Geoffrey O' Sullivan 

■ML. f if to f 2000 



Residence jig Suffolk St- flpt . g. H^O^ , N)Y jOOOZ 

Citizenship ^TVisio 



Post Office Address Same as residence address above 
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Full name of third joint inventor, if/52y Barclay A. Dunn 



Inventor's signature* /g^^ ^/M /^l/^ It / (?/^0&<? 

Residence IS IaJ <Sf #<fpA- ; A/W Y^^A// I '00 1 1 

Citizenship (^5/4** 

Post Office Address Same as residence address above 



[ ] ATTACHED IS/ARE ADDED PAGE(S) TO COMBINED DECLARATION AND POWER OF 
ATTORNEY FORM FOR SIGNATURE BY FOURTH AND SUBSEQUENT INVENTORS 

* Before signing this declaration, each person signing must: 

1 . Review the declaration and verify the correctness of all information therein; and 

2. Review the specification and the claims, including any amendments made to the claims. 

After the declaration is signed, the specification and claims are not to be altered. 
To the inventor(s): 

The following are cited in or pertinent to the declaration attached to the accompanying application: 



Title 37. Code of Federal Regulation. $ 1.56 

Duty to disclose information material to patentability. 

(a) A patent by its very nature is affect with a public interest. The public interest is best served, and 
the most effective patent examination occurs when, at the time an application is being examined, the Office 
is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that 
individual to be material to patentability as defined in this section. The duty to disclose information exists 
with respect to each pending claim until the claim is canceled or withdrawn from consideration, or the 
application becomes abandoned. Information material to the patentability of a claim that is canceled or 
withdrawn from consideration need not be submitted if the information is not material to the patentability 
of any claim remaining under consideration in the application. There is no duty to submit information 
which is not material to the patentability of any existing claim. The duty to disclose all information known 
to be material to patentability is deemed to be satisfied if all information known to be material to 
patentability of any claim issued in patent was cited by the Office or submitted to the Office in the manner 
prescribed by §§ 1 .97(b>(d) and 1 .98. However, no patent will be granted on an application in connection 
with which fraud on the Office was practiced or attempted or the duty of disclosure was violated through 
bad faith or intentional misconduct. The Office encourages applicants to carefully examine: 



(1) prior art cited in search reports of a foreign patent office in a counterpart application, and 
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(2) the closest information over which individuals associated with the filing or prosecution of 
a patent application believe any pending claim patentably defines, to make sure that any 
material information contained therein is disclosed to the Office. 

Title 35. U.S. Code 8 101 
Inventions patentable 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

Title 35 U.S. Code $ 102 

Conditions for patentability; novelty and loss of right to patent 
A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for patent, 

(b) the invention was patented or described in a printed publication in this or foreign country or in 
public use or on sale in this country, more than one year prior to the date of application for patent in the United 
States, or 

(c) he has abandoned the invention, or 

(d) the invention was first patented or caused to be patented, or was the subject of an inventor's 
certificate, by the applicant or his legal representatives or assigns in a foreign country prior to the date of the 
application for patent in this country on an application for patent or inventor's certificate field more than twelve 
months before the filing of the application in the United States, or 

(e) the invention was described in a patent granted on an application for patent by another filed in the 
United States before the invention thereof by the applicant for patent, or on an international application by another 
who has fulfilled the requirements of paragraphs (1), (2), and (4) of section 371(c) of this title before the invention 
thereof by the applicant for patent, or 

(f) he did not himself invent the subject matter sought to be patented, or 

(g) before the applicant's invention thereof the invention was made in this country by another had not 
abandoned, suppressed, or concealed it. In deterrmning priority of invention there shall be considered not only the 
respective dates of conception and reduction to practice of the invention, but also the reasonable diligence of one 
who was first to conceive and last to reduce to practice, from a time prior to conception by the other . . . 

Title 35. U.S. Code $ 103 

Conditions for patentability; non-obvious subject matter 

A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such 
that the subject matter as a whole would have been obvious at the time the invention was made to a person having 
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ordinary skill in the art to which said matter pertains. Patentability shall not be negatived by the manner in which 
the invention was made. 

Subject matter developed by another person, which qualifies as prior art only under subsection (f) or (g) of 
section 102 of this title, shall not preclude patentability under this section where the subject matter and the claimed 
invention were, at the time the invention was made, owned by the same person or subject to an obligation of 
assignment to the same person. 

Title 35. U.S. Code § 1 12 fin parri 
Specification 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise and exact terms also enable any person skilled in the art to which it 
pertains, or with which it is mostly nearly connected, to make and use the same, and shall set forth the best mode 
contemplated by the inventor of carrying out his invention. 

Title 35. U.S. Code $ 119 

Benefit of earlier filing date in foreign country; right of priority 

An application for patent for an invention filed in this country by any person who has, or whose legal 
representatives or assigns have, previously regularly filed an application for a patent for the same invention in a 
foreign country which affords similar privileges in the case of applications filed in the United States or to citizens of 
the United States, shall have the same effect as the same application would have if filed in this country on the date 
on which the application for patent for the same invention was first filed in such foreign country, if the application in 
this country is filed within twelve months from the earliest date on which such foreign application was filed; but no 
patent shall be granted on any application for patent for an invention which had been patented or described in a 
printed publication in any country more than one year before the date of he actual filing of the application in this 
country, or which had been in public use or on sale in this country more than one year prior to such filing. 

Title 35. U.S. Code § 120 

Benefit or earlier filing date in the United States 

An application for patent for an invention disclosed in the manner provided by the first paragraph of section 
1 12 of this title in an application previously filed in the United States, or as provided by section 363 of this title, 
which is filed by an inventor or inventors named in the previously filed application shall have the same effect, as to 
such invention, as though filed on the date of the prior application, if filed before the patenting or abandonment of or 
tennination of proceedings on the first application or an application similarly entitled to the benefit of the filing date 
of the first application and if it contains or is amended to contain a specific reference to the earlier filed application. 

Please read carefully before signing the Declaration attached to the accompanying Application. 

If you have any questions, please contact Morgan & Finnegan, L.L.P. 
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